Searching for Dependencies in Bayesian Classi ersMichael

Naive Bayesian classiiers which make independence assumptions perform remarkably well on some data sets but poorly on others. We explore ways to improve the Bayesian classiier by searching for dependencies among attributes. We propose and evaluate two algorithms for detecting dependencies among attributes and show that the backward sequential elimination and joining algorithm provides the most improvement over the naive Bayesian classiier. The domains on which the most improvement occurs are those domains on which the naive Bayesian classiier is signiicantly less accurate than a decision tree learner. This suggests that the attributes used in some common databases are not independent conditioned on the class and that the violations of the independence assumption that aaect the accuracy of the classiier can be detected from training data. The Bayesian classiier (Duda & Hart, 1973) is a probabilistic method for classiication. It can be used to determine the probability that an example j belongs to class C i given values of attributes of an example represented as a set of n nominally-valued attribute-value pairs of the form A 1 = V 1 j : If the attributes are independent, this probability is proportional to Equation 23.1 (23:1) Equation 23.1 is well suited for learning from data, since the probabilities ^ P(C i) and ^ P(A k = V k j jC i) may be estimated from the training data. To determine the most likely class of a test example, the probability of each class is computed with Equation 1. A classiier created in this manner is sometimes called a simple (Langley, 1993) or naive (Kononenko, 1990) Bayesian classiier. One important evaluation metric for machine learning methods is the predictive accuracy on unseen examples. This is measured by randomly selecting a subset of the examples in a database to use as training examples and reserving the remainder to be used as test examples. In the case of the simple Bayesian classiier, the training examples are used to estimate probabilities and Equation 23.1 is then used 1996 Springer-Verlag.